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1.0 


SUMMARY 


Military  cyber  operators  are  required  to  use  sustained  attention  or  vigilance  for  long  time  periods.  During 
this  time  they  encounter  lapses  in  attention  due  to  the  sometimes  monotonous  nature  of  their  tasks. 
Mistakes  during  these  tasks  can  have  serious  consequences  to  our  national  security.  The  purpose  of  this 
study  was  to  investigate  the  use  of  an  eye -tracker  to  detect  changes  in  vigilance  performance  during  a 
simulated  cyber  operator  task.  Twenty  participants  performed  four  sessions  of  a  40-minute  vigilance  task 
while  wearing  an  eye -tracker.  Blink  frequency,  blink  duration,  PERCLOS,  pupil  diameter,  pupil 
eccentricity,  pupil  velocity,  and  signal  detection  all  had  a  significant  change  over  time  (p<.05)  during  the 
vigilance  task.  The  significant  change  of  oculometric  measurements  indicates  that  oculometrics  could  be 
used  to  detect  changes  in  sustained  attention  for  cyber  operators.  Future  research  is  needed  to  assess  the 
real-time  effects  of  these  oculometrics  on  vigilance  performance,  especially  in  a  real-world  setting. 


2.0  INTRODUCTION 

Sustained  attention  and  vigilance  are  important  issues  in  today’s  military  cyber  operator  environment. 
Cyber  operators  are  required  to  monitor  several  screens  of  various  networks’  traffic  information  to  look 
for  specific  keywords,  internet  protocol  (IP)  addresses,  etc.  When  identified,  they  must  forward  the 
information  on  to  intelligence  services  for  further  analysis  (Lin,  2010).  Cyber  operators  experience  a  high 
degree  of  routine  with  few  incidents  while  monitoring  significantly  large  amounts  of  data  (Donald,  2008). 
This  type  of  work  requires  divided  attention,  selective  attention,  effective  scan  patterns,  sustained 
attention,  and  a  large  working  memory  capacity  (Donald,  2008).  Lapses  in  attention  can  cause  incidents 
to  go  unnoticed.  It  is  well  established  in  the  cognitive  performance  literature  that  operator  performance  on 
tasks  requiring  sustained  attention  or  vigilance  degrades  with  time;  this  is  known  as  the  “vigilance 
decrement”  (Hitchcock,  Warm,  Matthews,  Dember,  Shear,  Tripp,  Mayleben,  &  Parasuraman,  2003). 
During  the  vigilance  decrement,  critical  errors  are  made  that  can  have  severe  or  even  deadly 
consequences  (Hawley,  2006).  This  leads  to  an  increasing  need  to  employ  an  unobtrusive  way  to  monitor 
operator  vigilance  in  this  setting. 

These  sometimes  devastating  lapses  in  attention  are  possibly  due  to  the  monotonous  and  sometimes 
boring  nature  of  these  careers  (Frankmann  &  Adams,  1962;  Nachreiner  &  Hanecke,  1992).  Various  eye 
metrics  have  long  been  associated  with  arousal  levels.  Several  researchers  have  found  that  eye  tracking 
technologies  can  detect  fatigue,  boredom,  and  lapses  in  attention  (Dinges,  Mallis,  Maislin,  &  Powell, 
1998;  Russo  et  al.,  1999).  Specifically,  reduced  alertness  has  been  found  when  eye  blinks  are  longer  in 
duration  (Stem,  1999).  In  fact,  the  use  of  eye  tracking  technology  is  employed  regularly  in  the  trucking 
industry  to  monitor  driver  arousal  because  performance  becomes  less  consistent  and  vigilance 
deteriorates  as  a  person’s  sleepiness  increases  (Dinges,  1990).  Another  metric  that  provides  alertness 
information,  especially  in  the  trucking  industry,  is  PERCLOS  (percentage  of  eye  closure);  it  is  the  most 
widely  used  measure  of  real-time  alertness  in  this  industry  (Dinges  &  Grace,  1998;  Mallis  et  al.,  1999). 
However,  these  findings  are  the  result  of  studies  on  sleep-deprived  participants  who  are  not  tested  on 
vigilance  tasks. 

While  research  on  eye  tracking  in  regards  to  vigilance  is  not  as  extensive  as  in  the  fatigue  literature, 
studies  using  laboratory  vigilance  tasks  have  found  some  promising  results  using  oculometrics  to  detect 
attentional  levels.  Several  studies  have  indicated  that  eye  gaze  is  related  to  attention  (Blake  &  Sekuler, 
2006;  Kramer  &  McCarley,  2003;  Palmer,  1999);  therefore,  eye  movements  may  be  closely  related  to  our 
attentional  levels.  For  example,  some  have  found  that  well-rested  participants  who  are  placed  in  a  well-lit 
room  but  asked  to  do  a  boring  repetitive  task,  similar  to  our  task,  will  mimic  the  pupil  dilations  of  a  sleep 
deprived  individual  placed  in  a  dark  room  (Nishiyama,  Tanida,  Kusumi,  &  Hirata,  2007;  Warga,  Ludtke, 
Wilhelm,  &Wilhelm,  2009).  In  both  instances  the  pupils  dilate  initially  before  becoming  miotic 
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(Lowenstein,  Feinberg,  &  Lowenfeld,  1963;  Ludtke  et  al,  1998).  Beatty  (1982)  tested  this  finding  with  an 
auditory  vigilance  task  and  found  that  pupil  diameter  decreased  as  a  function  of  time-on-task.  In  a 
previous  study  we  found  that  the  oculometrics  of  blink  duration,  blink  frequency,  PERCLOS,  pupil 
diameter,  pupil  velocity,  and  pupil  eccentricity  could  be  indicators  of  vigilance  task  performance 
(Mclntire,  McKinley,  Goodyear,  Merrit,  Griffin,  Mclntire,  &  Bridges,  2011).  These  various  findings 
coupled  with  the  extensive  amount  of  research  with  sleep  deprived  individuals  lead  us  to  believe  that 
oculometrics  may  provide  a  reliable  method  for  assessing  operator  vigilance.  In  this  study  we  will  attempt 
to  replicate  our  previous  findings  using  a  more  real-world  relevant  task  that  requires  not  only  sustained 
attention  but  working  memory  and  divided  attention,  much  like  that  would  be  found  in  a  cyber  operators 
environment,  to  determine  if  the  similar  results  can  still  be  found  operationally. 


3.0  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 

3.1  Participants 

A  total  of  20  participants  (15  male,  5  female)  completed  this  study.  Male  and  female  civilian  and  active 
duty  participants  were  between  the  ages  of  21-41  years  (M  =  26.1).  Participants  were  compensated  $  10/hr 
for  their  time.  Participants  were  not  included  in  this  study  if  they  required  eyeglasses  for  vision  correction 
because  the  eye -tracker  was  mounted  on  eyeglass  frames.  The  use  of  contact  lenses  was  allowed. 

3.2  Equipment 
3.2.1  Eye-Tracker. 

Each  participant  was  required  to  wear  the  Eye -Com  (Reno,  NV)  alertness  monitoring  device  (see  Figure 
1)  during  the  vigilance  task  which  was  repeated  across  four  test  sessions,  each  on  different  days.  The 
device  consisted  of  two  infrared  (IR)-sensitive  cameras  and  a  linear  array  of  IR-illuminating  light 
emitting  diodes  (LEDs)  mounted  on  a  set  of  eyeglass  frames.  The  wavelength  of  the  LEDs  was  840  nm. 
The  cameras  were  angled  upward  toward  the  eyes  and  extracted  real-time  pupil  diameter,  eye-lid 
movement,  and  eye-ball  movement.  The  software  recorded  a  variety  of  measurements  including  eye- 
blink  duration,  eye-blink  frequency,  eye-blink  velocity,  percentage  of  time  the  eyes  are  closed 
(PERCLOS),  and  pupil  size.  The  sampling  frequency  of  this  device’s  data  recording  was  30  frames  per 
second.  The  tracker  monitored  blink  duration  and  frequency  by  tracking  the  occlusion  of  the  pupil.  When 
85%  of  the  pupil  was  occluded  by  the  eyelid  the  eyes  were  considered  closed  for  that  particular  frame.  To 
be  considered  a  “blink”  for  our  analysis  the  pupil  had  to  be  occluded  for  at  least  3  frames  because  median 
blink  rates  for  alert  individuals  usually  fall  between  130-170  milliseconds  (Schleicher,  Galley,  Briest,  & 
Galley,  2008).  If  there  were  less  than  3  frames  in  a  row  that  indicated  the  eyes  were  closed,  it  was  just 
considered  a  bad  data  point  where  the  tracker  lost  the  pupil  momentarily. 


2 

Distribution  A:  Approved  for  public  release;  distribution  unlimited. 
88  ABW  Cleared  5/2/2013;  88ABW-2013-2135 


Figure  1.  Eye-Com  Eye  Tracker 


PERCLOS  is  a  metric  that  is  calculated  by  the  eye -tracker  by  measuring  the  proportion  of  the  pupil  that  is 
occluded  by  the  upper  eyelid.  Therefore,  PERCLOS  is  the  proportional  amount  of  time  when  at  least  80% 
of  the  pupil  is  occluded  by  the  eyelid  in  a  1  minute  time  frame.  Pupil  velocity  is  calculated  by  tracking 
how  far  the  pupil  moved  in  a  single  frame  by  the  location  of  the  pupil  on  the  previous  frame.  Pupil 
eccentricity  is  the  extent  to  which  the  shape  of  the  pupil  deviated  from  being  circular.  The  shape  changes 
to  more  elliptical  due  to  occlusion  from  the  eyelid.  Therefore,  this  metric  is  more  a  function  of  the  nuance 
in  the  pupil  tracker  than  a  physiological  change  in  shape.  Elowever,  it  can  still  provide  valuable 
information  about  attention  if  it  also  reflects  information  about  the  eyelid  and  blinking. 

3.2.2  Vigilance  Task. 

Participants  performed  a  40-minute  vigilance  task  that  was  designed  to  simulate  a  cyber  defense 
operator’s  task.  The  Cyber  Defense  Task  (CDT)  was  developed  by  University  of  Dayton  Research 
Institute  (UDR1)  to  simulate  tasks  representative  of  those  found  in  Cyber  Defense  Operations.  The  task  is 
comprised  of  two  components  that  run  simultaneously.  The  first  is  a  textual  component  where  the 
participant  was  asked  to  monitor  and  respond  to  the  presence  of  a  suspicious  internet  protocol  (IP) 
address  and  port  combination  entering  the  network.  The  second  component  is  a  graphics  task  where  the 
participant  was  asked  to  monitor  and  respond  to  a  dangerous  increase  in  overall  traffic  within  the  network 
(Figure  2).  The  participant  was  asked  to  memorize  three  specific  IP  addresses  before  beginning  the  task. 
These  IP  addresses  were  suspicious  and  once  they  were  recognized,  the  participant  was  required  to  press 
a  key  on  the  keyboard  to  provide  an  alert  to  their  presence.  The  other  portion  of  the  task  showed  a 
simulated  2-D  graph  of  overall  IP  traffic  on  the  network.  Here,  the  participant  had  to  press  a  key  when 
the  IP  traffic  exceeded  a  predefined  limit  denoted  by  a  horizontal  red  line.  Both  the  textual  and  graphical 
task  had  a  critical  signal  event  rate  of  5%.  At  the  top  of  the  display  there  were  also  three  distracter  graphs. 
Participants  were  not  required  to  take  any  action  on  these  static  images.  Performance  efficiency  was 
assessed  in  terms  of  the  percentage  of  correct  signals  detected  (percent  hits).  This  variable  was  calculated 
every  1 0  minutes  over  the  continuous  40  minute  period.  Measuring  over  1 0-minute  epochs  was  conducted 
because  the  critical  signals  were  designed  to  appear  at  random  from  one  trial  to  the  next,  but  at  a  specific 
event  rate  within  10  minute  segments  of  trials  (5%).  A  new  stimulus  was  displayed  every  2000  msec  for 
the  graphical  portion  of  the  task  and  every  4000  msec  for  the  textual  portion.  These  rates  were  chosen 
based  on  piloting  study  results. 
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Figure  2.  Cyber  Defense  Task 


3.2.3  Vigilance  Tasks  are  very  Sensitive. 

Participants  were  run  in  a  room  isolated  from  any  noise  and  participants  were  required  to  wear  ear  plugs. 
Light  levels  were  maintained  to  be  consistent  throughout  the  experiment  and  glare  from  the  lights  onto 
the  task  screen  was  minimized  as  much  as  possible.  Participants  were  kept  away  from  any  possible 
distractions  including  being  able  to  see  the  experimenters.  In  this  study,  a  half  wall  was  used  to  isolate  the 
participant  from  the  experimenter.  The  experimenters  were  able  to  see  the  participant  and  what  they  were 
doing  but  the  participant  was  not  able  to  see  the  experimenters. 

3.3  Procedures 

No  study  specific  procedures  were  performed  without  a  written  and  signed  informed  consent  document. 
After  the  participant  was  consented  and  registered  into  the  study,  training  on  the  vigilance  task  began. 
Participants  received  a  verbal  briefing  and  PowerPoint  presentation  that  described  the  vigilance  task 
followed  by  two  5-minute  practice  sessions.  Also  during  this  time  the  participant  was  to  complete  a 
personality  and  brief  demographic  questionnaire.  Participants  then  donned  the  eye-tracker  and  completed 
the  40-minute  vigilance  task.  Afterwards,  the  participants  were  finished  for  that  day  and  returned  to  their 
normal  duties.  Each  participant  completed  four  data  collection  sessions.  Each  data  session  occurred  on  a 
separate  day.  During  each  data  collection  session,  they  repeated  the  procedures  of  the  initial  session 
except  for  the  personality  inventory  and  training  sessions. 

3.4  Data  Analysis 

Upon  completion  of  testing,  oculometric  data  and  vigilance  task  metrics  were  averaged  in  1 0-min 
increments  (10,  20,  30,  40  min)  and  used  as  dependent  variables  in  univariate  repeated-measures  analyses 
of  variance  (ANOVAs).  The  performance  data  from  the  dual  task  was  averaged  together  to  give  a  total 
score  for  each  time  epoch.  Factors  were  Day  (1  -  4)  and  Time  (10,  20,  30,  40  min).  Statistical 
significance  tests  were  based  on  an  alpha  level  of  .05.  Some  participants  had  a  day  of  performance  data 
that  was  not  usable  due  to  incorrect  data  recording.  During  post  processing  it  was  discovered  that  the 
CDT  had  one  IP  address  that  the  data  did  not  record  properly  causing  the  need  to  remove  that  day  of 
performance  from  the  data  analysis  if  the  participant’s  data  had  that  IP  address.  As  a  result,  a  participant’s 
data  was  not  included  in  analysis  unless  there  were  at  least  three  sessions  (days)  of  usable  data.  Proc 
Mixed  in  SAS  was  used  to  perform  the  repeated-measures  ANOVAs.  This  procedure  uses  maximum 
likelihood  to  estimate  covariances  within  a  subject  and  then  uses  these  covariance  estimates  to  estimate 
coefficients  for  fixed  effects.  Satterthwaite-type  degrees  of  freedom  were  used  for  all  F-tests  (SAS 
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Version  9.2).  Least  squares  means  (LSMeans)  are  means  adjusted  for  missing  data  and  were  used  since 
there  were  some  instances  where  the  eye -tracker  lost  the  pupil  during  data  collection. 


Next,  each  participant’s  days  were  categorized  as  either  a  Decrement  or  No  Decrement  day  depending  on 
the  percent  hits  performance.  If  the  linear  best-fit  slope  on  a  participant’s  percent  hits  across  the  four 
temporal  epochs  on  a  given  day  was  negative,  the  data  day  was  considered  as  a  Decrement.  Positive 
slopes  or  zero  +  0.1  slopes  were  considered  a  No  Decrement  day.  For  each  participant,  variables  were 
averaged  across  days  at  each  time  point,  separately  for  Decrement  and  No  Decrement  days.  Pearson 
partial  correlations  controlling  for  subject  (same  as  ANOCOV  with  subject  as  a  factor)  were  performed 
separately  for  Decrement  and  No  Decrement  days  to  relate  percent  hits  performance  to  the  six 
oculometrics. 


4.0  RESULTS  AND  DISCUSSION 

4.1  Results 

Results  are  presented  in  two  different  sections.  First,  we  present  the  results  for  the  Day  and  Time 
ANOVAs.  Second,  we  present  the  correlations  of  the  12  variables  in  relation  to  percent  hits.  Both 
analyses  included  all  20  participants. 

4.1.1  Analysis  for  Day  and  Time. 

The  repeated  measures  ANOVA  results  with  factors  Day  (1,  2,  3,  4)  and  Time  (10,  20,  30,  40  min)  are 
displayed  in  Table  1  below.  Significant  F-tests  have  the  p-value  cell  grayed.  ANOVAs  were  used  to 
compare  days  and  times  for  each  of  the  variables. 


Table  1.  ANOVA  Results  for  Day  and  Time 


Dependent  Variable 

1  Day 

j  Time 

Day*Time  | 

DF 

DFe 

F 

1 _ E _ 1 

DF 

DFe 

F 

p 

DF 

DFe 

i  F 

P 

Percent  Hits  (graph) 

3 

55.0 

6.58 

0.0593 

9 

219.0 

Imran™ 

0.5793 

Percent  Hits  (text) 

3 

56.2 

2.09 

wmw»M 

0.5933 

9 

166.2 

0.7651 

Left  Blink  Frequency  (blpm) 

mm 

55.9 

0.42 

|  0.7390 

■3 

57.4 

9.31 

0.0001 

9 

169.3 

1.87 

0.0596 

Right  Blink  Frequency  (blpm) 

3 

0.93 

3 

57.4 

6.34 

0.0009 

9 

168.9 

3.33 

0.0009 

Left  Blink  Duration  (ms) 

3 

2.09 

3 

57.1 

29.59 

0.0001 

9 

169.0 

1.06 

0.3944 

Right  Blink  Duration  (ms) 

3 

1.81 

3 

57.5 

25.88 

0.0001 

9 

169.3 

2.33 

0.0168 

Left  PERCLOS 

mm 

56.1 

0.33 

0.8005 

3 

57.4 

18.95 

0.0001 

9 

169.7 

1.64 

0.1070 

Right  PERCLOS 

3 

56.1 

0.62 

0.6030 

3 

57.5 

13.70 

0.0001 

9 

169.3 

1.80 

0.0706 

Left  Pupil  Diameter  (mm) 

3 

wm 

0.0060 

3 

57.6 

27.78 

0.0001 

9 

169.4 

1.77 

0.0780 

Right  Pupil  Diameter  (mm) 

3 

mm 

0.3666 

3 

56.1 

47.84 

0.0001 

9 

167.9 

2.64 

0.0070 

Left  Pupil  Eccentricity 

3 

0.64 

0.5942 

0.0001 

9 

168.6 

0.74 

0.6755 

Right  Pupil  Eccentricity 

3 

1.88 

0.1432 

0.0001 

9 

168.4 

0.67 

0.7364 

Left  Pupil  Velocity  (deg/s) 

3 

2.73 

0.0527 

0.0001 

9 

169.7 

0.78 

0.6307 

Right  Pupil  Velocity  (deg/s) 

3 

2.16 

0.1029 

EKHtrai 

0.0001 

9 

169.3 

0.84 

0.5843 

The  Day  of  data  collection  had  a  significant  effect  on  Percent  Flits  for  the  graphical  portion  of  the  task  as 
well  as  Left  Pupil  Diameter  (Table  1  &  Figure  4).  Percent  Flits  increased  as  the  participation  day 
progressed  (Figure  3).  On  Day  1  average  diameter  was  8.35  mm  (SEM  =  0.12)  and  Day  4  the  average 
diameter  for  the  left  eye  was  8.12  mm  (SEM  =  0. 12). 
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D  =  0.001  T  =  0.059  D*T  =  0.579 


D  =  0.006  T  =  0.001  D*T  =  0.078 


Time  on  task  had  a  significant  effect  on  Left  and  Right  Blink  Frequency,  Left  and  Right  Blink  Duration, 
Left  and  Right  PERCLOS,  Left  and  Right  Pupil  Diameter,  Left  and  Right  Pupil  Eccentricity,  and  Left 
and  Right  Pupil  Velocity.  As  the  Time  on  task  progressed.  Left  and  Right  Blink  Frequency  increased 
(Figure  5).  The  LSMean  across  the  four  days  for  the  first  10  minutes  of  the  task  was  17.1  blinks  per 
minute  (SEM  =  2.3)  in  the  left  eye  and  21.3  (SEM  =  2.3)  blinks  per  minute  for  the  final  10  minutes  of  the 
task.  The  right  eye  had  similar  results  with  16.0  blinks  per  minute  (SEM  =  2.2)  during  the  first  10  minutes 
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of  the  task  and  20.0  blinks  per  minute  (SEM  =  2.2)  for  the  final  10  minutes  of  the  task.  This  equates  to  a 
25%  increase  in  blinking  frequency  for  the  left  and  right  eye. 


D  =  0.739  T  =  0.001  D*T  =  0.060 


D  =  0.432  T  =  0.001  D*T  =  0.001 


Figure  5.  Left  and  Right  Eye  Blink  Frequency  Across  Time,  by  Day 


As  Time  on  task  increased  from  the  first  10  minutes  of  the  task  to  the  last  10  minutes  of  the  task,  Left  and 
Right  Blink  Durations  became  longer  (Figure  6).  The  LSMean  for  the  first  10  minutes  of  the  task  across 
the  four  days  was  196.8  ms  (SEM  =  7.8)  in  the  left  eye  and  191.7  ms  (SEM  =  5.1)  in  the  right  eye. 
Whereas,  the  LSMean  for  the  final  10  minutes  of  the  task  was  218.2  ms  (SEM  =  7.8)  in  the  left  eye  and 
210.4  ms  (SEM  =  5.1)  in  the  right  eye.  This  equates  to  11%  longer  blink  durations  in  the  left  eye  and  10% 
longer  blink  durations  in  the  right  eye. 


D  =  0.111  T  =  0.001  D*T  =  0.394  D  =  0.156  T  =  0.001  D*T=0.017 


Time  (min)  Time  (min) 

Figure  6.  Left  and  Right  Eye  Blink  Duration  Across  Time,  by  Day 


Left  and  Right  PERCLOS  increased  as  Time  on  the  task  increased  (Figure  7).  During  the  first  10  minutes 
of  the  task  PERCLOS  in  the  left  eye  had  an  LSMean  across  all  four  days  of  4.19%  (SEM  =  1.04)  and 
3.76%  (SEM  =  0.89)  in  the  right  eye.  During  the  last  10  minutes  of  the  task  the  LSMean  for  the  left  eye 
was  6.73%  (SEM  =  1.04)  and  5.81%  (SEM  =  0.89)  in  the  right  eye. 
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D  =  0.801  T  =  0.001  D*T  =  0.107 


D  =  0.603  T=  0.001  D*T  =  0.071 


Figure  7.  Left  and  Right  Eye  PERCLOS  Across  Time,  by  Day 


Time  on  task  also  had  a  significant  effect  on  Left  and  Right  Pupil  Diameter.  As  Time  on  the  task 
increased,  Pupil  Diameter  decreased  (Figure  8).  During  the  first  10  minutes  of  the  task,  the  LSMean  for 
Left  Pupil  Diameter  across  the  four  days  was  8.44  mm  (SEM  =  0.12)  and  Right  Pupil  Diameter  was  8.49 
mm  (SEM  =  0.13).  For  the  last  10  minutes  of  the  task  the  LSMean  for  Left  Pupil  Diameter  was  8.20  mm 
(SEM  =  0.13)  and  Right  Pupil  Diameter  was  8.22  mm  (SEM  =  0.13).  This  equates  to  a  3%  decrease  in 
pupil  diameter  for  both  eyes. 


D  =  0.006  T  =  0.001  D*T  =  0.078  D  =  0.367  T  =  0.001  D*T  =  0.007 


Time  (min)  Time  (min) 

Figure  8.  Left  and  Right  Eye  Pupil  Diameter  Across  Time,  by  Day 


Left  and  Right  Pupil  Eccentricity  increased  as  a  function  of  Time  on  task  increasing  (Figure  9).  During 
the  first  10  minutes  of  the  task  the  LSMean  across  the  four  days  in  the  Left  eye  was  0.524  (SEM  =  0.014) 
and  the  Right  eye  was  0.484  (SEM  =  0.013).  During  the  last  10  minutes  of  the  task  the  LSMean  in  the 
Left  eye  became  0.549  (SEM  =  0.014)  across  the  4  days  and  0.507  (SEM  =  0.013)  for  the  Right  eye.  This 
equates  to  a  5%  increase  in  both  eyes. 
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D  =  0.594  T  =  0.001  D*T  =  0.676 


D  =  0.143  T  =  0.001  D*T  =  0.736 


Figure  9.  Left  and  Right  Eye  Pupil  Eccentricity  Across  Time,  by  Day 


Time  on  task  also  had  a  significant  effect  on  Left  and  Right  Pupil  Velocity  (Figure  10).  Left  Pupil 
Velocity  during  the  first  10  minutes  of  the  task  had  an  LSMean  of  0.268  degrees  per  second  (SEM  = 
0.019)  across  the  4  days  and  an  LSMean  of  0.257  degrees  per  second  (SEM  =  0.019)  in  the  Right  eye. 
The  LSMean  for  the  last  10  minutes  of  the  task  in  the  Left  eye  was  0.351  degrees  per  second  (SEM  = 
0.019)  and  0.340  degrees  per  second  (SEM  =  0.019)  for  the  Right  eye.  This  equates  to  a  31%  increase  in 
saccadic  velocity  in  the  Left  eye  and  a  33%  increase  in  the  Right  eye. 


D  =  0.053  T  =  0.001  D*T  =  0.631  D  =  0.103  T  =  0.001  D*T  =  0.584 


Figure  10.  Left  and  Right  Eye  Pupil  Velocity  Across  Time,  by  Day 


An  interaction  effect  for  Day  and  Time  was  found  for  Right  Blink  Frequency,  Right  Blink  Duration,  and 
Right  Pupil  Diameter. 

4.1.2  Correlations. 

Pearson  partial  correlations  controlling  for  subject  were  performed  (separately  for  Decrement  and  No 
Decrement  days)  to  relate  Percent  Hits  to  the  eye  metric  variables.  Table  2  displays  the  correlations  with 
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their  corresponding  p-value.  Significant  partial  correlations  have  their  cells  grayed.  Percent  Hits  was 
combined  for  both  tasks  to  come  up  with  a  total  percent  hits  score.  If  there  was  a  decrease  in  performance 
in  either  task  or  both  tasks  that  day  was  considered  a  decrement  day. 


Table  2.  Pearson  Partial  Correlations  Controlling  for  Subject 


Variable  Correlated  With 

Percent  Hits 

Decrement 

No  Decrement 

r 

P 

r 

P 

Left  Blink  Frequency  (blpm) 

-0.29 

0.0329 

0.20 

0.1622 

Right  Blink  Frequency  (blpm) 

-0.20 

0.1337 

0.28 

0.0412 

Left  Blink  Duration  (ms) 

-0.43 

0.0011 

0.12 

0.4095 

Right  Blink  Duration  (ms) 

-0.36 

0.0064 

0.23 

0.1026 

Left  PERCLOS 

-0.40 

0.0023 

-0.07 

0.6216 

Right  PERCLOS 

-0.41 

0.0021 

0.06 

0.6482 

Left  Pupil  Diameter  (mm) 

0.53 

0.0001 

-0.07 

0.6220 

Right  Pupil  Diameter  (mm) 

0.55 

0.0001 

-0.14 

0.3136 

Left  Pupil  Eccentricity 

-0.50 

0.0001 

0.11 

0.4579 

Right  Pupil  Eccentricity 

-0.45 

0.0005 

0.09 

0.5128 

Left  Pupil  Velocity  (deg/s) 

-0.38 

0.0037 

0.33 

0.0186 

Right  Pupil  Velocity  (deg/s) 

-0.39 

0.0033 

0.33 

0.0175 

Left  Blink  Frequency  (Figure  1 1),  Left  and  Right  Blink  Duration  (Figure  12),  Left  and  Right  PERCLOS 
(Figure  3),  Left  and  Right  Pupil  Eccentricity  (Figure  14)  significantly  correlated  with  Percent  Hits  in  the 
Decrement  group  while  Right  Blink  Frequency  significantly  correlated  with  Percent  Hits  in  the  No 
Decrement  group.  As  performance  decreases  in  the  Decrement  group.  Left  Blink  Frequency,  Left  and 
Right  Blink  Duration,  and  Left  and  Right  PERCLOS  increased.  There  was  no  significant  relationship 
found  for  Blink  Duration,  PERCLOS,  Left  and  Right  Pupil  Eccentricity,  in  the  No  Decrement  group. 


Decrement  No  Decrement 


Left  r  =  -0.29  p  =  0.0329  Right  r  =  -0.20  p  =  0.1337  Left  r  =  0.20  p  =  0.1622  Right  r  =  0.28  p  =  0.0412 


O  =  Blink  Frequency 


Figure  11.  Mean  Percent  Hits  and  Left  and  Right  Eye  Blink  Frequency  Across  Time,  by  Day 
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Blink  Frequency  (blpm) 


Decrement 


No  Decrement 


Left  r  =  -0.43  p  =  0.0011  Right  r  =  -0.36  p  =  0.0064  Left  r  =  0.12  p  =  0.4095 


Right  r  =  0.23  p  =  0.1026 


t-lE 


>230 
1-220  3 
1-210  c 


f 


200  o 

3 

t190  'B 

- 

180 


10  20  30  40  10  20  30  40 

X  =  Percent  Hits  Time  (min) 


10  20  30  40  10  20  30  40 

Time  (min) 

O  =  Blink  Duration 

Figure  12.  Mean  Percent  Hits  and  Left  and  Right  Eye  Blink  Duration  Across  Time,  by  Day 


Decrement 


No  Decrement 


O  =  PERCLOS 

Figure  13.  Mean  Percent  Hits  and  Left  and  Right  Eye  PERCLOS  Across  Time,  by  Day 


Decrement  No  Decrement 

Left  r  =  -0.50  p  =  0.0001  Right  r  =  -0.45  p  =  0.0005  Left  r  =  0.1 1  p  =  0.4579  Right  r  =  0.09  p  =  0.5128 


Figure  14.  Mean  Percent  Hits  and  Left  and  Right  Eye  Pupil  Eccentricity  Across  Time,  by  Day 


Additionally,  Left  and  Right  Pupil  Diameter  were  significantly  correlated  with  Percent  Hits  in  the 
Decrement  group  (Figure  15).  As  performance  decreased  on  the  task,  Pupil  Diameter  decreased. 
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Pupil  Eccentricity 


Decrement 


No  Decrement 


Left  r  =  0.53  p  =  0.0001  Right  r  =  0.55  p  =  0.0001 


X  =  Percent  Hits  Time  (min) 

O  =  Pupil  Diameter 


Left  r  =  -0.07  p  =  0.6220  Right  r  =  -0.14  p  =  0.3136 


“0 

c 

■o 


D 


Q> 

3 


CD 

CD 


3 

3 


Figure  15.  Mean  Percent  Hits  and  Left  and  Right  Eye  Pupil  Diameter  Across  Time,  by  Day 


Left  and  Right  Pupil  Velocity  significantly  correlated  with  Percent  Hits  in  both  the  Decrement  and  No 
Decrement  group  (Figure  16).  In  the  Decrement  group,  Pupil  Velocity  increases  as  performance 
decreases.  In  the  No  Decrement  group,  performance  remains  fairly  stable  across  time.  Pupil  Velocity  of 
the  participants  in  this  group  increases  at  first  before  leveling  off. 


Decrement 

Left  r  =  -0.38  p  =  0.0037  Right  r  =  -0.39  p  =  0.0033 


No  Decrement 
Left  r  =  0.33  p  =  0.0186  Right  r  =  0.33  p  =  0.01 75 


X  =  Percent  Hits 
O  =  Pupil  Velocity 
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Figure  16.  Mean  Percent  Hits  and  Left  and  Right  Eye  Pupil  Velocity  Across  Time,  by  Day 


For  our  demographic  data  results  (Table  3),  we  found  that  the  age  of  the  participant  had  a  strong 
correlation  with  vigilance  performance.  Essentially,  younger  persons  were  more  likely  to  experience  the 
vigilance  decrement  on  any  given  day  relative  to  older  persons  (ages  ranged  from  19  to  41  years  with  an 
average  age  of  27  years,  n=36). 


Table  3.  Personality  and  Demographics  (asterisk  denotes  statistical  significance  at  an  alpha  level  of 
.05;  double  asterisks  denote  significance  at  an  alpha  level  of  .01) 


Measure 

Age 

Gender 

Early  Bird  /  Night  Owl 

Pearson’s  r 

-.35 

0.32 

0.403 

p-value  (two 
tailed) 

.036* 

.054 

.022** 

Sample  size 

36 

36 

32 
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We  also  collected  demographic  data  in  which  participants  self-classified  into  “Night  Owl”  types  (people 
who  prefer  to  stay  up  late  and  sleep  in)  versus  “Early  Birds”  (people  who  prefer  to  go  to  bed  early  and 
awaken  early).  We  found  a  very  strong  relationship  between  self-classification  of  Early  Birds/Night  Owls 
and  the  propensity  to  incur  a  vigilance  decrement  on  the  days  of  testing.  Specifically,  we  found  a 
significant  positive  correlation  (r  =  .403,  p[two-tailed]  =  .022)  suggesting  that  Night  Owls  are  much  more 
likely  to  demonstrate  the  vigilance  decrement  than  Early  Birds. 


4.2  Discussion 

The  discussion  section  is  broken  into  two  separate  parts,  first  the  discussion  on  the  eye  metrics  followed 
by  the  demographics  discussion. 

4.2.1  Oculometrics. 

Our  results  suggest  that  the  six  oculometrics  examined  in  our  study  correlated  with  changes  in 
performance  and  should  be  considered  in  applications  to  monitor  cyber  operator  vigilance.  These  include 
blink  frequency,  blink  duration,  PERCLOS,  pupil  diameter,  pupil  eccentricity,  and  pupil  velocity.  These 
metrics  could  be  beneficial  in  reducing  errors  and  improving  cyber  operator  performance. 

The  results  for  blink  frequency,  blink  duration,  and  PERCLOS  showed  that  as  attention  decreased 
(performance  declined)  there  was  an  increase  in  these  oculometrics.  This  indicates  that  poor  attention  to  a 
task  could  be  measured  by  an  increase  in  blink  rate,  longer  blink  durations,  and  a  longer  amount  of  time 
spent  with  the  eyes  closed.  We  found  this  same  effect  in  a  previous  study  using  a  less  complex  task 
(Mclntire  et  al.,  2011).  The  correlations  were  stronger  for  the  previous  study  in  all  eye  metrics  but  we 
believe  this  is  just  a  reflection  of  increased  variance  moving  away  from  a  strictly  controlled  laboratory 
task  to  a  more  real-world  relevant  task  with  more  potential  sources  of  variance  (Mclntire  et  ah,  2011).  It 
is  interesting  to  note  that  results  for  this  current  study  indicate  that  Right  Blink  Frequency  was  the  only 
metric  to  not  significantly  interact  with  Percent  Hits  for  the  Decrement  group  but  it  did  for  the  No 
Decrement  group.  In  the  previous  study,  our  results  indicated  that  performance  had  a  significant  effect  on 
blink  frequency  in  the  right  eye  only  for  the  decrement  group  (Mclntire  et  ah,  2011).  While  no  definitive 
explanation  can  be  offered  and  because  people  do  not  typically  blink  their  eyes  independently  of  one 
another,  we  should  note  our  observation  that  participants  would  completely  close  one  eye  and  attempt  to 
do  the  task  with  just  one  eye  open  as  the  task  progressed  probably  as  a  countermeasure  to  fight  task- 
induced  fatigue.  We  should  also  note  that  in  the  decrement  group,  both  eye  blink  frequency  measures 
were  negatively  correlated  with  performance  (although  only  the  left  eye  correlation  was  statistically 
significant).  The  opposite  pattern  occurred  in  the  no  decrement  group,  in  which  both  eye  blink  frequency 
measures  were  positively  correlated  with  performance  (again,  only  one  eye’s  correlation  was  significant). 
These  results  are  suggestive  of  potentially  low  statistical  power  or  small  effect  sizes,  which  may  have 
been  hampered  by  missing  data  and  by  splitting  our  overall  results  into  two  separate  analyses  (the 
decrement  versus  no  decrement  groupings).  Whatever  the  explanation  for  these  asymmetrical  findings 
across  the  eyes,  more  research  would  be  necessary  into  this  particular  eye  metric  before  it  could  be 
recommended  for  implementation  into  a  monitoring  system. 

Others  have  examined  blink  frequency  and  duration  during  a  vigilance  task  and  have  consistently  found 
an  increase  in  these  metrics  as  a  function  of  time -on-task  (Caipenter,  1984;  Funke,  2011;  Morris  & 

Miller,  1996;  Schroder  &  Holland,  1968).  Similarly,  Brookings,  Wilson  and  Swain  (1996)  found  that 
when  participants  were  paying  attention  and  concentrating  on  a  hard  high-workload  task  their  blink  rates 
would  decline  but  when  workload  levels  decreased  their  blink  rates  increased.  This  evidence  leads  us  to 
believe  this  metric  is  still  of  possible  use  with  more  research  on  more  operationally  relevant  tasks  and 
environments. 
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On  the  other  hand,  PERCLOS  appears  to  be  one  of  the  best  metrics  available  for  monitoring  vigilance 
according  to  this  study  and  our  previous  study  (Mclntire  et  al,  2011).  Specifically,  PERCLOS  negatively 
correlated  with  performance  and  appears  to  mimic  the  fluctuations  in  performance  for  the  Decrement 
group.  Other  research  also  indicates  that  PERCLOS  is  a  useful  indicator  of  performance  declines  induced 
by  time -on-task  fatigue  (Dinges  &  Grace,  1998).  Furthermore,  studies  have  also  shown  that  PERCLOS 
will  change  in  response  to  changes  in  cognitive  workload  (Kawashima,  O’Sullivan,  &  Roland,  1995; 
Marshall,  2007). 

The  one  oculometric  that  positively  correlated  with  performance  was  pupil  diameter.  Pupil  diameter 
decreased  as  the  number  of  critical  signals  detected  also  decreased.  When  pupil  diameter  is  small  the 
pupils  are  said  to  be  miotic.  Miosis  occurred  in  our  previous  study  as  well  as  other  studies  on  attention 
(Lowenstein,  Feinberg,  &  Lowenfeld,  1963;  Ludtke,  et  al.,  1998;  Mclntire  et  al.,  2011).  These  studies 
indicate  that  during  miosis  a  participant’s  performance  is  at  its  worst  (Nishiyama  et  al.,  2007;  Tsai  et  al., 
2007).  Therefore,  several  previous  studies  that  also  found  a  decrease  in  pupil  diameter  suggest  that  pupil 
diameter  may  be  an  indicator  of  poor  attention  (Nishiyama  et  al.,  2007;  Warga  et  al.,  2009),  which  is 
consistent  with  our  findings  from  both  studies. 

As  performance  on  the  task  decreased  an  increasing  pupil  velocity  was  found.  Because  our  pupil  velocity 
never  surpassed  3  degrees  per  second  our  observations  were  classified  as  microsaccades,  as  opposed  to 
the  larger  and  more  familiar  saccades.  This  is  consistent  with  findings  from  our  previous  study  (Mclntire 
et  al.,  2011).  We  believe  our  observations  are  unlikely  to  be  full  saccades  because  the  viewing  window 
for  the  critical  signals  is  so  small  that  full  saccades  are  not  necessary  to  perform  well  on  the  task. 

Saccadic  velocity  does  appear  to  be  related  to  attention  not  only  through  our  research  but  by  others  as 
well.  Galley  (1989)  found  that  tasks  requiring  high  levels  of  vigilance  increased  participant’s  saccadic 
velocity.  In  fact,  the  oculomotor  readiness  hypothesis  states  that  the  movement  that  controls  attention, 
fixation,  and  saccades  belongs  to  the  same  neural  circuitry  (Rizzolatti,  Riggio,  Dascola  &  Umilta,  1987); 
therefore,  attending  to  a  certain  location  should  result  in  faster  saccades  (Hoffman  &  Subramaniam, 

1995).  Our  results  of  increasing  microsaccades  with  time-on-task  could  be  indicating  that  the  participant 
is  trying  to  attend  to  the  task  more  because  they  are  aware  of  their  decreasing  arousal  levels.  It  is 
important  to  note  that  pupil  velocity  significantly  interacted  with  the  No  Decrement  group  as  well 
(although  the  correlations  were  in  opposite  directions  across  groups).  More  research  is  needed  into  this 
metric  to  determine  exactly  what  information  it  is  conveying  about  behavior. 

Pupil  eccentricity  was  found  to  increase  as  the  number  of  critical  signals  detected  decreased  for  the 
Decrement  group.  This  finding  is  also  concordant  with  our  previous  study  (Mclntire  et  al.,  2011).  Pupil 
eccentricity  is  increasingly  occurring  because  as  closure  of  the  eyes  as  indicated  by  blink  duration,  blink 
frequency,  and  PERCLOS  increases  with  time-on-task,  the  pupils  become  more  occluded  by  the  eyelids 
causing  their  shapes  to  appear  more  elliptical  than  round  to  the  eye  image  analysis  software  that 
calculates  their  shape  (Liu,  Sun,  &  Shen,  2010).  Furthermore,  Lowenstein  and  Loewenfeld  (1962)  believe 
that  pupil  eccentricity  is  an  indicator  of  arousal  levels.  In  general,  pupillary  activity  is  used  in  fatigue 
research  as  an  indicator  of  arousal  levels  because  pupillary  activity  is  considered  the  most  observable 
indicator  of  autonomic  nervous  system  activation  (Goldich,  Barkana,  Pras,  Zadok,  Hartstein,  &  Morad, 
2010).  Therefore,  our  findings  on  pupillary  activity  (pupil  diameter,  pupil  eccentricity,  and  pupil  velocity) 
coupled  with  findings  from  previous  research  lead  us  to  believe  that  a  good  system  to  monitor  sustained 
attention  should  include  monitoring  pupillary  activity. 

As  expected  for  our  experiment,  the  factor  of  Time  was  significant  for  all  oculometrics.  As  time -on-task 
progressed  blink  frequency  increased  25%,  blink  duration  increased  38%,  PERCLOS  increased  58%, 
pupil  diameter  decreased  3%,  pupil  eccentricity  increased  5%,  and  pupil  velocity  increased  32%  for  all 
subjects  and  sessions  (i.e.  not  broken  out  into  specific  groups).  These  changes  indicate  that  the  eyes  may 
reflect  the  changing  attention  levels  throughout  the  task.  The  factor  of  Day  also  had  a  significant  effect  on 
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Percent  Hits  for  the  graphical  portion  of  the  task.  Percent  Hits  increased  as  the  participation  day 
progressed  (Figure  3).  In  other  words,  participants  got  slightly  better  at  detecting  the  critical  signals  for 
this  portion  of  the  task  with  each  new  day  of  participation.  This  is  likely  to  be  a  training  or  practice  effect 
that  is  commonly  observed  in  repeated  measures  experiments. 

4.2.2  Demographics. 

For  our  demographic  data  results,  we  found  that  the  age  of  the  participant  had  a  strong  correlation  with 
vigilance  performance.  Essentially,  younger  persons  were  more  likely  to  experience  the  vigilance 
decrement  on  any  given  day  relative  to  older  persons.  Although  previous  research  has  also  claimed  a 
relationship  between  gender  and  fatigue,  specifically  with  men  being  more  resilient  to  the  effects  of 
fatigue  (DeVries  &  Van  Heck,  2002),  thus  suggesting  a  possible  similar  relationship  with  vigilance,  we 
found  no  compelling  evidence  either  way  although  the  trend  favored  women.  Admittedly,  this  finding  is 
just  fractionally  non-significant  potentially  due  to  our  sample  size  being  rather  small  and  skewed  to  over¬ 
representation  of  males  with  about  20%  female  volunteers;  or  7  female  participants  in  our  current 
combined  sample  size  of  36,  so  this  trend  might  well  disappear  with  further  research. 

We  also  collected  some  demographic  data  in  which  participants  self-classified  into  “Night  Owl”  types 
(people  who  prefer  to  stay  up  late  and  sleep  in)  versus  “Early  Birds”  (people  who  prefer  to  go  to  bed  early 
and  awaken  early).  We  found  a  very  strong  relationship  between  self-classification  of  Early  Birds/Night 
Owls  and  the  propensity  to  incur  a  vigilance  decrement  on  the  days  of  testing.  Specifically,  we  found  that 
Night  Owls  are  much  more  likely  to  demonstrate  the  vigilance  decrement  than  Early  Birds.  All 
participants’  data  collection  was  done  during  normal  business  hours,  many  of  which  occurred  in  the 
morning  hours,  which  could  suggest  that  Night  Owls  might  not  have  been  fully  awake  (perhaps  near  a 
low  point  in  their  circadian  cycle).  Future  research  on  this  topic  might  find  it  useful  to  record  time  of  day 
of  each  session,  the  number  of  hours  of  sleep  the  few  days  before,  sleeping  and  rising  times,  subjective 
ratings  of  sleep  quality,  etc.  It  might  also  be  interesting  to  investigate  the  extent  to  which  self-classified 
“Early  Birds”  are  actually  able  to  go  to  bed  late  and  sleep-in  versus  their  actual  sleep  times. 


5.0  CONCLUSION 

Our  results  indicate  that  changes  in  oculometrics  correspond  to  changes  in  vigilance  performance  during 
a  cyber  operator  task.  These  results  are  reflective  of  previous  findings  using  a  potentially  less 
operationally  relevant  task  (Mclntire  et  al.,  2011).  Therefore,  it  appears  that  moving  toward  a  more 
operationally  relevant  task  is  possible  and  that  field  testing  is  necessary  and  possible  using  this  method. 
Implementation  of  an  eye-tracking  device  to  monitor  the  attention  of  cyber  operators  may  significantly 
decrease  human  error  in  the  future  and  possibly  avoid  devastating  consequences  of  vigilance  lapses 
through  the  use  of  alerting  perceptual  cues  or  non-invasive  brain  stimulation  techniques  to  augment 
human  performance.  Future  research  is  needed  to  assess  these  oculometrics  in  real-time  and  to  assess 
their  effectiveness  in  the  field.  Also,  the  wearable  eye -tracker  used  in  this  study  proved  to  lend  itself  well 
to  a  more  operational  environment  compared  to  off-body  trackers  although  the  most  useful  tracker  in  an 
operational  setting  would  need  to  be  wireless. 
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LIST  OF  ABBREVIATIONS  AND  ACRONYMS 


ANOVA 

Analyses  of  Variance 

CDT 

Cyber  Defense  Task 

LED 

Light  Emitting  diode 

IP 

Internet  Protcol 

IR 

Infared 

LSMeans 

Least  Squares  Means 

PERCOS 

Percentage  of  Eye  Closure 

SAS 

Statistical  Analysis  Software 

SEM 

Standard  Error  of  the  Mean 

UDDI 

University  of  Dayton  Research 
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